The present invention relates to the field of information indexing, cataloguing and retrieval, and in particular to a system and method for automatically cataloguing Internet information repositories, creating an eXtensible Markup Language (XML) metaindex in an encoded XML format (i.e., the Resource Description Framework (RDF) format), and providing a mechanism to effectively search and retrieve the information.
In the last few years, there has been an explosion of information available on the Internet. In the very early 1990s, the Internet was a network consisting of computers from military, research, and educational organizations. There were small collections of information available through mostly file transfer protocol (FTP) sites and Gopher sites. With the advent of the web and increases in bandwidths beginning in about 1993s, people began to put more and more information on the Internet.
Originally, the Internet was only available for non-commercial research and educational use. When the Defense Advanced Research Projects Agency (DARPA) relaxed usage restrictions, finally permitting commercial use, Internet usage exploded. Today, most households have Internet access and anyone with Internet access can publish information on the Internet.
Shortly after the advent of the web, users realized that there was a need to have a way to search the Internet to assist users in locating information. Websites such as Lycos(trademark) and AltaVista(trademark) were developed to meet this need. These sites used spiders to scan the Internet for content, collecting and indexing keywords. These full-text-based indexes were then used on a website to assist users in searching the Internet to locate needed information. This method was effective when the Internet was young. Recognizing problems associated with large quantities of indexes, many larger search engine sites, such as Yahoo(trademark) and Excite(trademark), began to manually catalogue the indexed material. Manual cataloguing is not an effective methodology for organizing the vast amount of information on the WWW.
Today, most of the available content is unstructured so that it is difficult to locate pertinent data. Anyone with Internet access can publish any information they wish on the Internet. As the cost of access and disk space has decreased, the volume of information available has grown tremendously. Elementary search engines that simply create indexes of keywords are becoming increasingly ineffective in identifying relevant documents. There is a growing need for more effective search systems.
There is an additional need to provide a search system that can be used to perform a search across many heterogeneous information retrieval systems. For example, many organizations have built information retrieval systems to permit users to obtain documents published by that organization. It is desirable to provide a search system that can index and catalogue information stored in many different formats on different websites, permitting users to perform a search through a single web portal.
Finally, there is a need to provide a system for performing automated cataloguing and indexing of documents. Prior art systems have simply created keyword indexes. There is a need for a system that uses a thesaurus and a classification system to determine both keywords for an indexed document but also a class for the document to permit more effective search and retrieval of information.
As the quantity of information available on the Internet grows, it is becoming more and more important to provide more advanced search and retrieval capabilities. Keyword indexing alone is proving inadequate in providing a search system that permits a user to effectively locate and access information on the Internet.